To solve the problem that traditional image classification methods deal with the whole image in a non-hierarchical way, an image classification method based on visual saliency detection was proposed. Firstly, the visual attention model was employed to generate the salient region. Secondly, the texture feature and time signature feature of the image were extracted by Gabor filter and pulse coupled neural network, respectively. Finally, the support vector machine was adopted to accomplish image classification according to the features of the salient region. The experimental results show that the image classification precision rates of the proposed method in SIMPLIcity and Caltech are 94.26% and 95.43%, respectively. Obviously, saliency detection and efficient image feature extraction are significant to image classification.